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Abstract 



In order to quantitatively study object perception, be it perception by biological systems or by machines, one needs to create objects and object 
categories with precisely definable, preferably naturalistic, properties 1 . Furthermore, for studies on perceptual learning, it is useful to create novel 
objects and object categories (or object classes) with such properties 2 . 

Many innovative and useful methods currently exist for creating novel objects and object categories 3 " 6 (also see refs. 7,8). However, generally 
speaking, the existing methods have three broad types of shortcomings. 

First, shape variations are generally imposed by the experimenter 5 ' 9 ' 10 , and may therefore be different from the variability in natural categories, 
and optimized for a particular recognition algorithm. It would be desirable to have the variations arise independently of the externally imposed 
constraints. 

Second, the existing methods have difficulty capturing the shape complexity of natural objects 11 " 13 . If the goal is to study natural object 
perception, it is desirable for objects and object categories to be naturalistic, so as to avoid possible confounds and special cases. 

Third, it is generally hard to quantitatively measure the available information in the stimuli created by conventional methods. It would be desirable 
to create objects and object categories where the available information can be precisely measured and, where necessary, systematically 
manipulated (or 'tuned'). This allows one to formulate the underlying object recognition tasks in quantitative terms. 

Here we describe a set of algorithms, or methods, that meet all three of the above criteria. Virtual morphogenesis (VM) creates novel, naturalistic 
virtual 3-D objects called 'digital embryos' by simulating the biological process of embryogenesis 14 . Virtual phylogenesis (VP) creates novel, 
naturalistic object categories by simulating the evolutionary process of natural selection 9 ' 12 ' 13 . Objects and object categories created by these 
simulations can be further manipulated by various morphing methods to generate systematic variations of shape characteristics 15 ' 16 . The 
VP and morphing methods can also be applied, in principle, to novel virtual objects other than digital embryos, or to virtual versions of real- 
world objects 9 ' 13 . Virtual objects created in this fashion can be rendered as visual images using a conventional graphical toolkit, with desired 
manipulations of surface texture, illumination, size, viewpoint and background. The virtual objects can also be 'printed' as haptic objects using a 
conventional 3-D prototyper. 

We also describe some implementations of these computational algorithms to help illustrate the potential utility of the algorithms. It is important 
to distinguish the algorithms from their implementations. The implementations are demonstrations offered solely as a 'proof of principle' of 
the underlying algorithms. It is important to note that, in general, an implementation of a computational algorithm often has limitations that the 
algorithm itself does not have. 

Together, these methods represent a set of powerful and flexible tools for studying object recognition and perceptual learning by biological 
and computational systems alike. With appropriate extensions, these methods may also prove useful in the study of morphogenesis and 
phylogenesis. 
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Video Link 



The video component of this article can be found at http://www.jove.com/video/3358/ 



Protocol 



1. Creating Naturalistic Virtual 3-D Objects using VM 

1- To create digital embryos, use the Digital Embryo Workshop (DEW; see Table 1). Each run generates a single embryo 14 , the shape of which 
is unique to a given set of settings (or 'genotype') used for the given run (Figure 1). The 'cells' of the embryo are represented as triangles 14 . 

1 . Run the program as many times as needed to generate the desired number of embryos. 

2. If more complex shapes are desired, increase the number of growth cycles, i.e. the number of times the cells of the embryo will divide. 
Note that this will also slow down the program. If it is necessary to create virtual objects other than digital embryos, use commercially 
available 3-D modeling tools or obtain virtual objects from commercial vendors (Table 1). 

2. It is generally advisable to save the virtual objects in a commonly used file format, such as OBJ, so that the objects can be readily imported 
into a commercial 3-D modeling toolkit. To this end, the DEW writes objects in OBJ format by default. 

3. Visual stimuli can be generated using one or more digital embryos using a 3-D modeling and rendering environment (Table 1). Use standard 
graphical operations such as varying the orientation, size, lighting, surface texture and background to create the desired stimuli (see Figure 
2). 

2. Creating Naturalistic Object Categories using VP 

1 . To generate object categories, generate descendants (or 'children') of the given ancestor (or 'parent') object using a desired combination of 
the processes in Step 1.1 above (Figure 3) 9 ' 10 ' 12 ' 13 . 

1 . Some methods described below for creating smooth shape variations, such as morphing or principal components (see Steps 3 and 4 
below), work better if all the input objects have the same number of cells and if there is one-to-one correspondence among the vertices 
of the objects. For creating such objects, use only those VM processes that will not change the number of cells and will preserve the 
one-to-one correspondence of vertices among the objects (see, e.g., generations G 2 to G 3 in Figure 3). For instance, cell division 

and programmed cell death alter the number of cells, and make it much harder (although not impossible 17 ' 18 ) to determine one-to-one 
correspondence between the vertices of a given pair of objects. 

Note that the processes that alter the number of cells in a given object also alter its shape complexity. In general, the greater the 
number of cells, the greater the shape complexity of the object and smoother its surface. 

2. If necessary, virtual objects other than digital embryos can be used as inputs to VP (Figure 4). 

2- The objects within a given category can be further selected so as to achieve a given distribution of features 19 . For instance, one can 
selectively eliminate mid-sized objects from a given category in order to generate a bimodal distribution of object size. 

3. There is no single method that is universally optimal for measuring the available shape information for all categories, nor is there a single 
method that is optimal for categorizing all objects 20 " 22 . Thus, the experimentor must choose these methods based on the categories and 
computational goals at hand 20 " 22 . Step 4 describes a commonly used method for manipulating various aspects of available shape information. 

4- The similarity between a given pair of categories can be objectively measured using available phylogenetic methods 23 ' 24 . For instance, the 
vertical (or 'evolutionary') distance between a given pair of categories, as measured by hierarchical clustering methods in the R statistical 
toolkit, is an objective measure of category similarity 25 ' 26 . 

3. Additional Methods of Creating Shape Variation: Digital Morphing 

1 . Given any pair of objects so that each vertex of one object corresponds to exactly one vertex of the other object (i.e., objects with one-to- 

17 18 27 29 

one correspondence between vertices), morphing is straightforward ' ' " : In this case, smooth variations (or 'morphs') between the two 
objects are produced by smoothly interpolating between the corresponding vertices and normals (Figure 5). Depending on the pair of objects 
chosen, morphing will result in new categories or additional children within a category. 

1 ■ The objects shown in Figure 5 were created using linear morphing 27 " 29 . The objects can be morphed (or warped) by a vast array of 

other available deformation techniques 17 ' 18 . 

2. To create a desired distribution of morphed shapes, choose the interpolation points accordingly. 

4. Additional Methods of Creating Shape Variation: Principal Components 

1 ■ In order to use principal components to generate shape variations, one needs to first determine the principal components 15 . Principal 
components are specific to the given set of objects used for determining them 26 . For good results, use at least 30 objects with one-to-one 
correspondence between vertices 26 . 
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2. Generate an average object from a desired set of n input objects, by separately averaging the coordinates and the normal of each vertex 
across all objects. Thus, the x coordinate of a given vertex k of should be the average of the x coordinates of vertex k of all n objects, and so 
forth. 

3. Use the Matlab function princomp to determine principal components of the n objects. This will generate n-1 non-zero eigenvectors, along 
with the corresponding n-1 eigenvalues 26 . 

4. To generate a new object Aj from a given principal component P, , multiply P, by the corresponding eigenvalue Aj and a desired weight Wj and 
add to the average object: 

Aj= +w j A i P i 

5. Each unique Wj will generate a unique object. By smoothly varying w, one can create smooth shape variations along a given principal 
component. 

6. To create shape variations along an independent shape dimension, repeat step 4.4 using a different principal component. 

7. To create a desired distribution of shapes along a given principal component, use the desired distribution of w. 

8. To create a multi-dimensional grid of shapes, use a set of weights for each of several principal components: 

A M = A+w)A l P l +w 2 k ?L 2 P 2 +... 

5. Creating Haptic Versions of 3-D Objects 

1 . 'Print-out' 3-D objects using a 3-D prototyper (or 3-D 'printer'). If necessary, adjust the object's size and smooth the object's surface prior to 
printing. 

6. An Exemplar Application: Bayesian Inference of Image Category 

1 . An important task in visual processing is inferring the category to which a given observed object belongs. Although the exact mechanism 

9 12 13 30 32 

of this inference is unknown, there is both computational and physiological evidence ' ' ' that it involves using the information about 
known features of the object in the given image to infer the category of the object. Here, we will illustrate how this inferential process may 
work in a Bayesian framework, and how digital embryos may be useful for research in this area. 

2. For simplicity, we will assume that the categorization task is binary and involves distinguishing category Kfrom category L (Figure 3). Let C 
be the category variable. We will infer that C = Kor C = L according to whether the observed image / belongs to category Kor /_. A typical 
approach to categorization involves: 

1. Computing the probability that the category is K given the information in the image, denoted p(C = K |l); 

2. Computing the probability that the category is L given the information in the image, denoted p(C = L | /); and 

3. Picking the category with the higher probability. 

3. Next, we will assume for simplicity that there is exactly one binary feature F. This feature may be either present in the image (denoted F = 1) 
or absent from the image (denoted F = 0). This example will use the 'informative fragment' feature shown in Figure 8. Informative fragments 
were first described by Ullman and colleagues 33 . In the present case, we will use the image template shown in Figure 8 as the feature, and 
a threshold value of 0.69. To determine whether this feature is present in a given image (say, the rightmost image in row G3 in Figure 3), we 
will use the following steps: 

1 . Slide this template over all possible locations in the image and compute, at each location, the absolute value of normalized cross- 
correlation between the template and the underlying sub-image. 

2. Select the image location with the highest value (0.60 in the present case). 

3. If this value is above the threshold, conclude that the feature is present; otherwise, conclude that it is absent. In our case, since the 
highest correlation 0.60 is below the threshold of 0.69, we conclude that the feature is absent in this image. 

4. The rationale of using such features, and the mechanisms of selecting features and determining their thresholds are beyond the scope 
of this report, but are described in detail in refs. 33, 30. 

4. Within the framework of feature-based inference, we assume that all the information the observer extracts from the image is contained in the 
value of this feature, /'. e., that p(C | /) = p(C | F). 

Therefore, the task becomes that of determining the value of F in the given image (present or absent), computing p(C = K \ F) and p(C = L \ 
F) for that value F, and selecting the category with the higher probability. 

5. In the Bayesian framework, 

P{C[F)= P(C)P(F\C) 
P(F) 

Therefore, 

p(C-k\f)-X c -™ fic -V 

P(F) 
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p(C = L\ F) = 



p(C = L)p(F\C = L) 
P(F) 



K | F) and p(C = L \ F), it is not necessary to compute 



Note that the denominator in the two equations is the same. Therefore, to compare p(C : 
the denominator; rather, it is sufficient to compute the quantities 

p(C=K\F) # p(C=K)p(F\C=K) 

and 

p(C=L\F)#p(C=L)p(F\C=L) 

These are sometimes called 'unnormalized probabilities'. The term p(C) is called 'prior', and the term p(F | C) is called 'likelihood'. 

6. For simplicity, we will assume a 'flat' prior: p(C = K) = p(C = L) = 0.5. 

7. The task is now to compute p(F\C), the probability of a given feature value in an image of a given category C. 

1 . We will use the six images of category L (Figure 3) as an example to compute p(F = 1 |C = /_), the probability that the feature is present 
in an image of category Z_. 

2. To compute p(F = 1 |C = L), first take all the training images that belong to L. Again, these are shown in Figure 3. 

3. For each image, determine whether the feature value is 1 (present) or 0 (absent), as described in (6.3. 1 )-(6.3.3). In our case, for the six 
images in Figure 3, the values are: [0, 0, 1, 0, 1, 0]. 

4. Compute the fraction of images in which the feature value is 1 . In our case, it is 2/6 = 0.33. 

5. Therefore, p(F = 1 |C = L) = 0.33. Note that to obtain accurate estimates, one should use at least 30 images per class. 

6. In a similar manner, we can compute that p(F = 0|C = L) = 0.67, p(F = 1 |C = K) = 0.83, p(F = 0|C = K) = 0.17. 

8. Given these values, inference can be performed. Assume that we are given a new image (Figure 9), and the task is to determine its category 
label. This is performed as follows: 

1 . We determine whether the feature F is present in the image, as described in (6.3.1 )-(6.3.3). In our case, the feature is present, so F = 
1. 

2. Using equations (1 ) and (2), and the values computed in (6.7.5)-(6.7.6), we can calculate that p(C = K \ F = 1 ) = 0.42 and p(C = L \ F = 
1) = 0.17. 

3. Given this information, it is rational to conclude that the image is from category K, with relatively low confidence. 

9. It could be interesting and useful to test such an approach in a biological system (say, psychophysical^). In this case, behavioral predictions 
could be made and compared against the computational results such as those obtained in (6.8.2). To make accurate predictions, good 
estimates of p(F |C) are necessary. Such estimates, however, may be difficult to obtain for familiar objects. The reason is that a subject's 
estimate of p(F |C) is not observable directly, and it is extremely difficult to compute it since the subject's prior exposure to natural images is 
uncontrollable and unknown. In contrast, if digital embryos are used for the experiments, then it is possible to precisely control which embryos 
and embryo categories the subject is exposed to. This makes it easy to compute the quantities of interest from exactly the same data the 
subject was exposed to. The fact that embryos' appearance is naturalistic is helpful, since it makes it likely that the subject is using the same 
categorization strategy as with natural categories. 



Representative Results 



VM can be used to generate a virtually limitless supply of novel 3-D shapes. Some exemplar digital embryos generated using the VM algorithm 
are shown in the bottom panel of Figure 1. Each of these 16 embryos were generated by using the 'growEmbryos.exe' program in the Digital 
embryo tools for Cygwin (see Table 1 ) for 40 growth cycles. All other growth parameters were set internally by the program. Most of these 
parameters were constant (i.e., identical from one embryo to the next). A few parameters such as the location and strength of morphogen 
sources, were random parameters set internally by the program independently for each run. The shape variations among these 16 embryos 
arose solely as a result of the variations in these random parameters. 

Some examples of surface texturing 34 ' 35 using some arbitrarily chosen textures are shown in Figure 2A. Visual scenes of arbitrary complexity 
can be created using a commercially available 3-D modeling and rendering environment, as shown in Figure 2B. 

A representative 'family tree' generated by VP using digital embryos is shown in Figure 3. Comparable trees can also be constructed using 
objects other than digital embryos, as shown in Figure 4. Note that in either case, the objects that share a common ancestor straightforwardly 
constitute a category, although the experimenter may also choose to define a category as any other set of objects. It is worth noting from Figure 
4 that our current implementation of the VM and VP algorithms tends to produce relatively smooth, curved surfaces, as opposed to jagged or flat 
objects. It is also worth noting that this is presumably the limitation of our implementation of these algorithms and not the algorithms themselves, 
since biolological processes can produce objects with flat surfaces and jagged outlines (e.g., rose leaf). 

Figures 5 and 6 illustrate the typical results of two methods that can be used in addition to, or instead of, VP for creating principled variations in 
object shape and object categories. 

The top panel of Figure 7 illustrates visual renderings of two digital embryos, and the bottom panel of Figure 7 illustrates the corresponding 
printouts generated by a commercially available 3-D prototyper. 

Figures 8 and 9 illustrate the procedures described in Section 6 for using image fragments to categorize a given visual object. 
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Figure 1. Virtual morphogenesis. The bottom panel illustrates a type of novel, naturalistic, virtual 3-D objects called "digital embryos" 14 . 
Digital embryos can be generated by simulating one or more of some of the key processes of biological embryogenesis: morphogen-mediated 
cell division, cell growth, cell movement and programmed cell death 7 ' 8 ' 36 ' 37 . Each run starts with an icosahedron (shown in the top panel), 
and generates a unique embryo, depending on the VM parameter settings (or the 'genotype') of that embryo. Thus, the 16 embryos in the 
bottom panel have different shapes, because they all have different genotypes. Note that simpler or more complex shapes can be generated 
as needed (e.g., to optimally stimulate neurons at a given level of the visual hierarchy) by manipulating the genotype of the embryo. All of 
the aforementioned embryogenetic processes except programmed cell death were simulated in generating the embryos shown. Simulated 
programmed cell death is especially useful for creating targeted indentations (not shown). 
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Figure 2. Creating visual stimuli using digital embryos. Like any virtual 3-D object, digital embryos can be graphically manipulated to create 
visual scenes of arbitrary complexity using any standard 3-D graphical toolkit. This figure illustrates some common manipulations. (A) The same 
digital embryo is textured using many different textures, and lit from an invisible light source at top left. (B) A camouflaged scene is created by 
resizing and re-orienting the digital embryo and digitally placing it against the same background it was textured with. The digital embryo can be 
found in 'plain view' in the lower right quadrant. For additional examples of visual stimuli created using digital embryos, see refs. 9,10,12-14,38. 
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Figure 3. Creating digital embryo categories using VP. The VP algorithm emulates biological evolution, in that in both cases, novel objects 
and object categories emerge as heritable variations accumulate selectively. At each generation Gj , selected embryos procreate, leading to 
generation G i+1 . The progeny inherit the shape characteristics of their parent, but accrue shape variations of their own (as determined by small 
variations in their genotype) as they develop. This figure shows a 'family tree' of three generations of descendants starting from a single common 
ancestor, an icosahedron. Note that, in this case, the shape complexity increases from the icosahedron to generation G-i, but not from G 1 
onward. This is because increase in cell numbers (i.e., cell division) was allowed from the icosahedron to generation G-i, but not from G 1 onward. 
In general, cell division tends to increase shape complexity, whereas other morphogenetic processes such as cell movement and cell growth 
change shape without changing the overall complexity of the shape. 



Figure 4. VP using virtual objects other than digital embryos. This figure helps illustrate the general principle that virtual objects other than 
digital embryos can be used as input to VP. The VP algorithm in its current form can operate on any virtual 3-D object whose surface consists 
solely of triangles. Generation G 1 comprised of (from left to right) a gourd, diamond, face mask, apple, rock, and cactus. Note that the objects in 
generation G-i in this figure do not have a common ancestor, because VP does not require it. Objects in G 2 and G 3 represent the descendants of 
the rock in G-|. No cell divisions were allowed in any generation, so that all shape variations arose solely from the movement and/or growth of the 
individual 'cells' of the given object. 
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Figure 5. Using morphing to create smooth variations in shape. Morphing involves taking two given objects (far left and far right embryo 
in this figure) and calculating the intermediate objects (intervening embryos) by interpolating between the corresponding vertices of the two 
designated objects. In the case shown, all vertices were interpolated using the same scalar factor, resulting in a linear morphing. However, it 
is also possible to morph the objects non-linearly (not shown). Morphing is computationally straightforward when there is an exact one-to-one 
correspondence between the vertices of two objects, as in the case shown. However it is possible, in principle, to morph between any two given 
virtual objects regardless of whether their vertices correspond exactly, although there is no unique principled method fordoing so 17 ' 18 . 

% 

«%%%%* \ \\ \ 
%%%%%%%%%% 

Figure 6. Using principal components to create smooth variations in shape. (A) Average embryo. This embryo represents the arithmetic 
average of 400 embryos (200 each from categories K and L in Figure 3). Principal components were calculated as described in Step 4.3. 
Note that principal components represent mutually independent, abstract shape dimensions of the 400 embryos (not shown) 25 ' 26 . 400 embryos 
yield 399 non-zero principal components 25 ' 26 , which together account for all the variance, or the shape information, available collectively in the 
embryos. By convention, principal components are arranged in the decreasing order of their eigenvalues, or the proportion of the overall variance 
they explain 25 ' 26 . In this case, the first two principal components respectively accounted for 73% and 19% of the shape information available in 
the 400 embryos. (B) Embryos that represent different weights (or more precisely, weighted eigenvalues) of Principal Component 1 . The weights 
varied from +2 (far left) to -2 (far right) in equal steps of -0.2. (C) Embryos that represent different weights of Principal Component 2. The weights 
also varied from +2 (far left) to -2 (far right) in equal steps of -0.2. Note that manipulating principal components does not exclusively manipulate 
any given specific body part of the embryo (e.g., the wings of the embryo in the case shown). However, if necessary, body parts of virtual 3- 
D objects can be manipulated in any arbitrary user-defined fashion using most of the commercially available 3-D modeling environments (not 
shown). 
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Figure 7. Creating haptic objects. Virtual 3-D objects can be 'printed' as haptic objects using a standard, commercially available 3-D 'printer' 
or prototyper. This figure shows digital embryos rendered as visual objects (top row) or as the corresponding haptic objects (bottom row). The 
haptic objects shown in this figure were printed to be about 6 cm wide (scale bar = 1 cm), although the objects can be printed at much smaller or 
larger sizes. 



Figure 8. A template for an example informative fragment. In this example, the template has a threshold of 0.69 associated with it. 




Figure 9. A new image for which the object category is not known and needs to be determined. 



Discussion 



Usefulness of VM and VP in Cognitive Science Research 

We have previously described the usefulness of VM and VP in detail ' ' ~ . Briefly, VM, especially the digital embryo methodology, is useful 
because it provides a principled and flexible method for creating novel, but naturalistic 3-D objects 14 . Similarly, VP provides a principled method 
of creating naturalistic categories 9 ' 10 ' 12 ' 13 . It is worth noting that object categories generated by VP share many features with object categories 
in nature, including the fact that the categories tend to be hierarchical in nature, and the feature variations within and across categories arise 
independently of the experimenter and the algorithms for classifying them 39 . 
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Current Limitations and Future Directions 

Three current limitations of our protocol and the directions for future work they suggest are particularly noteworthy: First, both VM and VP 
simulate biological processes. While we show that non-biological virtual objects can be used as substrates for these processes, the underlying 
processes are still biologically motivated. However, natural objects - biological and non-biological alike - undergo shape changes due to non- 
biological forces. For instance, rocks may change in shape due to geological processes such as erosion or sedimentation. New categories of 
rock may arise from other such geological processes. It should be relatively straightforward to incorporate these processes into the repertoire of 
available shape change algorithms. 

The second major limitation of our protocol is that its current repertoire of dynamic shape changes is rather limited. It is desirable to incorporate a 
larger array of shape changes, such as biological motion, or motion due to external forces such as wind, water or gravity. We expect that it will be 
relatively straightforward to bring to bear the known computer animation algorithms to implement such dynamic shape changes. 

The third major limitation of our protocol is that VM currently does not include many other known morphogenetic processes including, most 
notably, gastrulation 36 . It also fails to incorporate some known constraints, such as the fact that morphogenesis in plants is mediated entirely by 
growth, with little or no cell movement possible, because of the cell walls 36 . Similarly, VP does not include other known phylogenetic processes 
such as genetic drift 40 . Addressing these limitations would help greatly facilitate the use of our protocol in developmental, ecological and 
evolutionary simulations. 
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